Persona Based Evaluation of Search Systems

“Search evaluation is notoriously complex, with even minor tweaks to hyperparameters or embeddings creating widespread ripple effects across retrieved results. Mitigating this inherent “instability” in search algorithm changes has long been a challenge. Traditional approaches, such as composing test cases (e.g., as supported by Quepid), offer a degree of control and consistency. However, writing and maintaining test cases is a painstaking task, particularly in dynamic environments where catalogs are constantly updated with new items.

Our proposed method introduces a groundbreaking solution: leveraging user modeling inspired by the “LLM as a judge” paradigm to automate query and result-set generation. This approach dynamically creates realistic query-result pairs by simulating diverse user personas, each designed to evaluate different modalities (image, video, audio) within the catalog under varying prompts.

The innovation lies in the adaptability and efficiency of the system:

• Dynamic Test Coverage: Personas adapt to catalog changes, ensuring that new items are immediately evaluated without manual intervention.
• Multi-Modality Testing: By supporting multiple content types, the system mirrors the diversity of modern search use cases.
• Fully Automated Pipeline: The process eliminates the need for manually written test cases, reducing overhead and accelerating iteration cycles.

This talk will provide an in-depth look at the methodology, the benefits of dynamic user modeling, and real-world results from applying this system. Attendees will leave with actionable insights into transforming their search evaluation strategies, unlocking new levels of stability and precision in their algorithms.”

Uri Goren

Argmax

Uri Goren is the CEO and Chief Data Scientist of Argmax, a boutique AI consulting firm specializing in machine learning, natural language processing, and recommendation systems. With over 15 years of experience in data science, Uri has led AI-driven projects across multiple industries, including fintech, cybersecurity, healthcare, and e-commerce. Before establishing Argmax, Uri co-founded BestPractix, an AI-driven company designed to assist legal professionals in accelerating the contract drafting process. In 2020, BestPractix was acquired by Litera Systems. Under his leadership, Argmax has delivered cutting-edge AI solutions for leading companies such as Visa, Unity, Minute Media, and various high-growth startups. Uri is known in Israel for his weekly podcast Explainable that dives into machine learning concepts and trending news.